1. Technical Field
The invention relates to a method for determining a weight coefficient for an artificial associative neuron synapse and a corresponding artificial associative neuron synapse. The invention particularly relates to implementing the synapse in such an artificial associative neuron in which the weight coefficient, or strength, of the synapse is determined on the basis of a temporal average of the product of two signals according to the Hebb rule or a modification thereof (Jain, Anil K., Mao Jianchang, Mohiuddin K. M. (1996) “Artificial Neural Networks: A Tutorial” in Computer, March 1996, p. 31 to 44).
2. Discussion of Related Art
International patent application WO 98/43159 discloses an artificial associative neuron that can be used in artificial neural networks. The neurons are connected in parallel to form layers, and such layers are arranged in succession to form a neural network. Neural networks can be used in various calculation and pattern recognition tasks.
FIG. 1 illustrates a prior art associative neuron including n synapses. Since the number n may be quite high (ranging from 100 to 10 000, for example), FIG. 1 shows only the first, second and nth synapse of the neuron. The input signals of the neuron are a main signal s and associative signals a1, a2, . . . an. Label so indicates the output signal of the neuron. Said signals are, for example, voltage levels. Circuit blocks 1 to 4, 8 to 11 and 12 to 15 correspondingly describe the first, second and nth synapse of the neuron, and its operation is described in greater detail below. Block 5 is a summing circuit in which the output signals obtained from the synapses are summed. Block 6 is a summing circuit in which the main signal s and the sum of the output signals obtained from the synapses are summed. Block 7 is a threshold circuit.
Information about the simultaneous occurrence of the main signal s arriving at the associative neuron and the associative signal ai (where i=1,2 . . . ,n) arriving at the synapse is stored in the synapses. This information may be presented, for example, in voltage form so that the voltage concerned represents the temporal average of the product s*ai of the main signal and the associative signal. For this purpose the synapse includes a multiplier block (blocks 1, 8, 12) in which said product is calculated (s*ai). In FIG. 1, signals s and a1 are applied to the first synapse in order to calculate said product, and the product s*a1 is calculated in the multiplier block 1. Signals s and a2 are applied to the second synapse, and the product s*a2 is calculated in the multiplier block 8 of the second synapse, and signals s and an are applied to the nth synapse, and the product s*an is calculated in the multiplier block 12 of the nth synapse.
If the signals s and ai can have only the logical values 0 and 1 (for example, a particular supply voltage value corresponds to the logical value 1 and the zero level corresponds to the logical value 0), then said product (s*ai) can be calculated as a logical product. The logical product (s*ai) is applied to blocks 2 (first synapse), 9 (second synapse) and 13 (n synapse), in which the temporal average of said product is formed and maintained. The temporal average is conveyed from blocks 2, 9 and 13 to blocks 3, 10 and 14, respectively.
In blocks 3, 10 and 14 the temporal average is compared to a particular threshold level. When the temporal average exceeds the threshold level, then said associative signal ai can be considered to correlate with the main signal s. Thus, a value deviating from zero, i.e. number 1, is set as the actual strength of the synapse or as the weight coefficient, the value otherwise being zero. The desired weight coefficient of the synapse wi, where i=1,2, . . . ,n depending on the synapse concerned, is the output of blocks 3, 10 and 14. For example, the weight coefficient w1 of the first synapse is the output of block 3, the weight coefficient w2 of the second synapse is the output of block 10, and the weight coefficient wn of the nth synapse is the output of block 14. The weight coefficient of the synapse obtains value 0 or 1.
The weight coefficients of the first, second and nth synapse are applied to blocks 4, 11 and 15, respectively, in which the weighted product wi* ai is calculated in order to obtain the output signals of the synapses. Said output signals are applied to block 5, where they are summed, as mentioned above. The sum is applied to block 6, in which the main signal s is summed with the sum of the output signals obtained from the synapses calculated in block 5. The output signal of block 6 is applied to block 7. Block 7 is a threshold circuit that determines the output signal so of the entire neuron by comparing the sum calculated in block 6 to a particular threshold voltage. The signal so is thus the output signal of block 7, which is simultaneously the output signal so of the entire neuron.
A problem with such an associative neuron synapse is the technical implementation of the generation and maintenance of the temporal average of the product of said two signals (s*ai).
FIG. 2 shows a known implementation that is applicable for determining the weight coefficient of each associative neuron synapse shown in FIG. 1. This known implementation is a circuit having said two signals as the input signals thereof, i.e. the main signal s and the associative signal ai. The output signal of the circuit is the weight coefficient wi of the synapse. The subindex i may obtain values 1 to n depending on which synapse weight coefficient is being determined.
The main signal s and the associative signal ai are applied to an AND circuit, AND1, where the logical product of the signals s and ai is formed. The AND1 circuit implements the multiplier block 1, 8 or 12 of the synapse shown in FIG. 1. As both the main signal s and the associative signal ai have a logical value 1, the output of AND1 circuit obtains value 1, whereas it otherwise obtains value zero. As noted above, the logical value 1 may correspond to a particular voltage value of a signal (such as 3.3 V) and the logical value 0 may correspond to the voltage value zero, for example.
The logical product, or the AND1 circuit output, is applied through a diode D1 and a resistor R11 to a temporal averaging RC circuit that comprises a resistor R21 and a capacitor C11. The resistor R21 and capacitor C11 are connected from one end to the ground level and from the other end to the other end of the resistor R11. As the logical product is 1, the output of the AND1 circuit is nearly the same as the supply voltage of the AND circuit, in which case the capacitor C11 is charged towards the supply voltage with the time constantτ, where τ=R11 * C11
In the time constant equation R11 is the resistance of the resistor R11 and C11 is the capacitance of the capacitor C11. The time constant equation holds true when the resistor R21 is much higher than the resistor R11, in which case the. effect of the resistor R21 on the time constant can be left unnoticed.
When the logical product is 0, the output of the AND1 circuit goes to the zero level, whereby the capacitor C11 is discharged (partly or entirely depending on the discharge time) through the resistor R21, until the logical product again obtains the value 1, whereby the charging state of the capacitor C11 starts to increase again. The diode D1 prevents the discharging of the capacitor C11 through the resistor R11 and the AND1 circuit. Thus the capacitor C11 can be discharged through the resistor R21 only with the time constantτ, where τ=R21*C11
In the time constant equation R21 is the resistance of the resistor R21 and C11 is the capacitance of the capacitor C11.
The capacitor C11 is charged and discharged as a temporal function of the logical product s*ai. The temporal average of the logical product is then stored in the capacitor and is instantaneously represented by a voltage across the capacitor C11. The diode D1, the resistors R11 and R21 as well as the capacitor C11 thus implement block 2, 9 or 13 shown in FIG. 1.
The voltage across the capacitor C11 is applied to the first input of a voltage comparator COMP1. The input impedance of the voltage comparator COMP1 is high so that the capacitor C11 cannot be discharged through here. A particular reference voltage V is applied to the second input of the voltage comparator COMP1. If the signals s and ai simultaneously obtain value 1 sufficiently many times so that the voltage of the capacitor C11 exceeds the reference voltage of the voltage comparator COMP1, i.e. the so-called learning threshold of the synapse, then the output voltage of the voltage comparator COMP1 rises up.
The output of the voltage comparator COMP1 is applied to a set reset circuit SR1. A reset signal generally provided with value 0 is the second input in the set reset circuit SR1. The set reset circuit SR1 comprises two outputs Q and Q′, one of which being inverted. The value of the output Q is the weight coefficient wi of the synapse. When the output voltage of the voltage comparator COMP1 increases, the set reset circuit SR1 permanently changes its state from value 0 to value 1, so that the output Q, i.e. the weight coefficient wi of the synapse, obtains the value 1. The voltage comparator COMP1 and the set reset circuit SR1 thus implement block 3, 10 or 14 shown in FIG. 1 for the generation of the weight coefficient wi of the synapse.
A microcircuit implementation of the RC circuit shown in FIG. 2 presents a problem, as the required time constants call for high capacitance values. However, in practice only fairly small capacitors, with capacitances of 1 pF or less, can realistically be placed onto the microcircuit. If the capacitor C11 is of this size, then the resistor R21 must correspondingly have an extensively high value (typically 1 TΩ or higher), for the desired time constant. The implementation of such a high resistance on the microcircuit is nevertheless difficult. In addition, a capacitor of the magnitude 1 pF takes up a lot of space on the microcircuit.